ACCURACY

11.26 Accuracy is the proximity of an estimate to some notional true value. This is an important component of quality as it relates to how well the data portray reality, which has clear implications for how useful and meaningful the data will be for interpretation or further analysis. In practice, the true value is rarely able to be determined and independent sources, if available, are also likely to include errors and omissions. For this reason it is not possible to comprehensively assess, in a quantitative manner, the accuracy of international merchandise trade statistics.

11.27 The following assessment of accuracy involves:

• a brief review of the major sources of inaccurate data
• a subjective assessment of exports and imports at varying commodity levels
• a revisions analysis of international merchandise trade statistics.

SCOPE AND COVERAGE DEFICIENCIES

11.28 The statistics are compiled from export and import declarations and are not collected by sample survey so they are not subject to sampling errors. However, they are subject to reporter error. While the DIBP apply various logical and legality checks and are able to audit business records they operate in a self-regulation environment which mainly targets declarations with perceived risk.

11.29 There has been significant growth in the volume of imports with values below the imports customs value threshold. Goods valued $1,000 or less are excluded from international merchandise trade statistics. According to the final report of the Low Value Parcel Processing Taskforce the volume of low value goods arriving by air cargo increased by 58% (to $4 million) over the period 2008-09 to 2010-11. While significantly less goods arrive through the international mail stream, there has been a large increase in the volume of these (up by 104% to $40 million between 2006-07 and 2010-11). Most goods arriving by mail are estimated to have values below the imports customs value threshold (see pages 29-33 in Low Value Parcel Processing Taskforce - Final Report July 2012. The number and value of export transactions which are excluded from the statistics are much lower despite the higher export declaration threshold ($2,000).

11.30 For details regarding the difference between scope (economic territory) and coverage (statistical territory), refer to Table 2.1 in Coverage.


Input and output processing

11.31 The ABS has implemented best practice input and output processing systems, processes and procedures to ensure the quality of international merchandise trade statistics. These include:

• Quality strategies to ensure the data is fit for purpose e.g. edit checks, significance editing, data confrontation with other sources.
• In cases where initial unconfirmed estimates of data provided by exporters who have been accorded Confirming Exporter Status by the DIBP are identified as inconsistent with confirmed data, careful estimates based on previous trends are made. Such estimates replace unconfirmed data until confirmed data are provided.
• Testing and evaluation plans prior to the introduction of any new or updated processes/and or systems to ensure errors and potential quality issues are identified and resolved.
• Training and development of staff to ensure they have the skills and abilities to identify key statistical issues and potential quality impacts.
• Risk mitigation and assessment processes, e.g. quality gates and data management information to ensure all data are received, loaded, processed and accounted for at each processing stage.

Methodology

11.32 Methodological changes can affect the comparability of data before and after the changes are implemented but they are generally directed towards improving the overall quality of the statistics. Historically there have been a small number of significant methodological changes to international merchandise trade statistics. In the last 25 years the most significant changes occurred in 1988 with the introduction of the HS and in 1992 when exports were converted from a processing date to a departure date basis. Both these changes were significant at the point of introduction but represented major improvements to the quality and international comparability of the statistics.

11.33 Wherever possible the ABS attempts to assist users to understand changes to methods or classifications by:

• notifying changes in advance
• preparing and releasing information about the impact of changes
• preparing correspondences showing the links between the new and old classifications
• providing information on both bases for a limited period or less frequently using look-up tables to re-classify information to the previous classification or method.

11.34 The accuracy and accessibility of international merchandise trade statistics is affected by confidentiality restrictions. International merchandise trade statistics are subject to confidentiality restrictions which means statistics are able to be released unless, and until, an individual or organisation demonstrates that disclosure would be likely to enable the identification of that person or organisation. Where a request has been made to confidentialise data, the full detailed dataset is not available to users, but the value of confidential data is included in total exports and imports.


Valuation issues

11.35 By carefully checking all large value transactions, most cases of significant over-valuation error are detected and corrected by the ABS, prior to publication of the statistics. Significant under-valuation and smaller value inaccuracies may not be detected by the DIBP and ABS quality checking procedures and may therefore contribute to inaccuracies in the statistics.

11.36 For imports the DIBP system uses official exchange rates to convert values at the time of export from the overseas country of export, this minimises currency conversion errors. Exporters are required to declare goods prior to export and these are converted to Australian dollars by the ABS from RBA data.

11.37 Exporters may undertake currency conversions using a different exchange rate, or one applying on a different day to that at the time of exportation. This includes practices such as hedging which are designed to offset adverse currency movements. However, as the majority of exports are negotiated in one of the reportable currencies, any inappropriate conversion practices in the remaining exports will have a very limited impact on the statistics.


SUBJECTIVE ACCURACY RATINGS

11.38 The accuracy of international merchandise trade statistics can also be assessed by assigning subjective accuracy ratings at different HS levels. Subjective accuracy ratings are not derived by a particularly rigorous process, but represent an intuitive assessment by international merchandise trade compilers taking into account knowledge about input errors, impressions about the coverage and reliability of source data and amendment records received.

11.39 Table 11.2 assesses accuracy in terms of commodity values for exports and imports at the HS 2, 4, 6 and 8 (exports only) and 10 (imports only) digit level. These levels have been chosen because international merchandise trade statistics are disseminated and used at these levels, the data are internationally comparable up to the 6 digit level and program resources are focussed on ensuring the quality of these data for macroeconomic purposes. Each component has been assigned one of the following grades:

• very good
• good
• fair
• poor.

AHECC (Exports)		CUSTOMS TARIFF (Imports)
Level	Commodity value	Level	Commodity value
2 digit	good	2 digit	good
4 digit	good	4 digit	good
6 digit	good	6 digit	good
8 digit	fair	10 digit	good

11.40 The ratings given in Table 11.2 for exports and imports are the same except that the most detailed (10 digit) imports statistics have a higher rating than the 8 digit export statistics. The quality of import statistics has historically been considered better than export statistics because of the scrutiny provided by the DIBP to fulfil their community protection and revenue collection roles. The absence of very good ratings reflects the trade off between ensuring highly accurate statistics, disseminating timely monthly international merchandise trade statistics and limited resources.


REVISIONS ANALYSIS

11.41 A revisions analysis for the period July 2008 to June 2012 (48 months) is included as another way to assess the quality of international merchandise trade statistics. The analysis measures the mean bias and dispersion between the average initial monthly published result for the period and the average final published result.

11.42 Bias is a measure of the extent to which initial estimates are lower or higher than the final estimate and gives an indication of the direction of revisions. Unscaled bias is calculated as the average of the differences, over the observation period, between the initial estimate of a category for each month and the latest available estimate for the same month, with positive and negative revisions being netted against each other. This measure can be described as the average of the values of all revisions taking account of sign. Scaled bias is calculated by expressing the average revision for each category as a percentage of the average initial value of that category for the month, and then calculating the average of these percentage revisions.

11.43 Dispersion is a measure of the 'spread' between the initial and final estimates and indicates the magnitude of revisions. Unscaled dispersion is calculated as the average of the differences, over the observation period, between the initial estimate of a category for each month and the latest available estimate for the same month, however positive and negative revisions are not netted against each other. This measure can be described as the average of the absolute values of all revisions, without regard to sign. Scaled dispersion is calculated as the average of the absolute value of the percentage revisions.

11.44 As measurements of dispersion are based on absolute values, the value of dispersion measures is generally larger than the equivalent measure of bias. Only if all revisions were in the same direction, would the results be the same for both measures. The larger the difference between the absolute values of the two measures for a particular series, the greater the variability in the direction of revisions.


ANALYSIS OF EXPORT STATISTICS

11.45 Table 11.3 shows that the revisions to initial published monthly export estimates had on average, a positive bias of $490 million or 2.21% for the period July 2012 to June 2014. Negatively biased revisions indicate that the initial estimate tends to overstate the latest estimate. The dispersion measures were generally higher, indicating that the individual monthly revisions were both positive and negative. The average magnitude of revisions to total exports, without regard to sign, was also $490 million or 2.21% of the average monthly estimate.

11.46 The main contributors to the negatively biased revisions were crude materials, inedible, except fuels (SITC 2) and mineral fuels, lubricants and related materials (SITC 3). The unscaled and scaled dispersion measures were also higher for these SITC sections due to:

• relatively higher values and a greater contribution to total exports of the commodities included in these SITC sections
• at the time of initial reporting to the DIBP the quantity and value information for these commodities is estimated and revised in later months when the actual transaction values are available. This contributed significantly to revisions when:
  • the timing of export contracts for these commodities changed from annual to quarterly and then monthly
  • the exchange rate depreciated significantly
  • commodity prices, in particular iron ore depreciated significantly.

11.47 As crude material prices dropped in 2014, reporters initial estimates to the DIBP were often overstated. In response, the ABS began applying an adjustment to the originally reported data, which was subject to revision, based on historical trends. The figures are revised in subsequent months, once actual transaction values are available. This method yields significantly more accurate results in comparison to solely relying on administrative DIBP data, at both aggregate and transactional levels. These changes are not reflected in the current analysis as they were only implemented in January 2015. However, post January 2015 outputs have shown a significant reduction in data revisions.

11.48 The average revisions for all other SITC sections were downward as shown by the negative bias measures in Table 11.3.

SITC	Average initial published estimate	Mean Revisions
		Bias		Dispersion
		Unscaled	Scaled	Unscaled	Scaled
	$m	$m	%	$m	%
0	2 334.04	-17.17	-0.74	19.60	0.84
1	178.83	-1.71	-0.96	1.72	0.96
2	8 474.13	-330.83	-3.90	334.05	3.94
3	5 823.96	-76.83	-1.32	77.83	1.34
4	51.25	-0.33	-0.64	0.95	1.85
5	657.21	-2.42	-0.37	4.02	0.61
6	1 216.63	-21.29	-1.75	21.30	1.75
7	1 149.04	-12.75	-1.11	12.76	1.11
8	419.38	-4.29	-1.02	4.67	1.11
9	1 847.50	-21.92	-1.19	22.31	1.21
Total	22 151.96	-489.54	-2.21	489.53	2.21

ANALYSIS OF IMPORT STATISTICS

11.49 Table 11.4 shows that the revisions to initial published monthly import estimates had, on average, a negative bias of $118 million or 0.58% for the period July 2012 to June 2014, indicating that the initial estimate overstated the final estimate. This pattern is also reflected in the revisions at the SITC section level. The dispersion measures were slightly higher, indicating that the individual monthly revisions were both positive and negative.

SITC	Average initial published estimate	Mean Revisions
		Bias		Dispersion
		Unscaled	Scaled	Unscaled	Scaled
	$m	$m	%	$m	%
0	888.67	2.56	0.29	2.60	0.29
1	198.50	0.92	0.46	1.50	0.76
2	208.67	0.37	0.18	0.80	0.38
3	3 531.63	-36.03	-1.02	37.80	1.07
4	45.88	0.15	0.33	0.50	1.09
5	2 054.92	-10.66	-0.52	11.20	0.55
6	2 257.79	4.15	0.18	6.30	0.28
7	7 765.29	-47.82	-0.62	48.70	0.63
8	2 639.33	-15.49	-0.59	16.00	0.61
9	856.67	-16.06	-1.87	20.30	2.37
Total	20 447.33	-117.91	-0.58	117.90	0.58